Study 1 


Efficacy of the Social Skills Improvement System - 
Classwide Intervention Program (SSIS - CIP) in the Primary Grades 


Authors and Affiliations: 

James C. DiPerna, PhD 

The Pennsylvania State University 

Puiwa Lei, PhD 

The Pennsylvania State University 

Jill Bellinger, PhD 

The Pennsylvania State University 

Weiyi Cheng, MS 

The Pennsylvania State University 


SREE Spring 2014 Conference Abstract Template 



Abstract Body 


Background 

Teaching children to get along with others, care about themselves, and actively 
participate in learning are three of the most important outcomes of the schooling process 
(Gresham & Elliott, 1990; Lane, Givner, & Pierson, 2004; Wentzel, 1993). Yet children in some 
schools are not achieving these outcomes, and many educators have not received adequate 
training to create instructional environments that facilitate these outcomes. As a consequence, 
some schools have become places where children feel uncomfortable, unsupported, and 
ultimately, uninterested in learning. Under these conditions, learning is negatively affected even 
for some of the most able students. The importance of this problem has been acknowledged for 
decades in major reports such as Nation at Risk (1983) and federal initiatives under educational 
refonn agendas like Goals 2000. The development of socially competent students also has been a 
top concern echoed in surveys of parents, teachers, and students as well as other educational 
stakeholders (e.g., Lickona, 1991). 

The genesis of ineffective instructional environments and underachieving students is 
complex and varies across individuals and schools/communities. Consequently, the solutions to 
improving learning, preventing or reducing inappropriate behavior, and facilitating the prosocial 
development of students require multifaceted and sustained interventions. The Social Skills 
Improvement System (SSIS) is a comprehensive program that integrates multiple levels of 
assessment and intervention to improve children’s social skills and engagement in classroom 
learning. The Classwide Intervention Program (SSIS-CIP) is the universal component of the 
SSIS, and it has been developed to help students learn the 10 social skills that teachers have 
identified as most critical to classroom success (Elliott & Gresham, 2007). This is accomplished 
through the implementation of a 10-week structured curriculum that includes teacher lesson 
plans, video vignettes, student workbook activities, progress monitoring, and parent 
communications. Although the SSIS-CIP is based on empirical evidence from the social skills 
literature and is commercially available, its efficacy in school settings has yet to be detennined. 

Focus of Study 

The purpose of this study was to evaluate the efficacy of the SSIS-CIP using a Multi-Site 
Cluster Randomized Trial (CRT). Specifically, we hypothesized that children in classrooms 
implementing SSIS-CIP would demonstrate improved social skills compared to children in non- 
implementing (business-as-usual) classrooms. In addition, children in SSIS-CIP classrooms were 
expected to demonstrate increased academic engagement during classroom instruction. 

Setting 

The study was conducted in two Pennsylvania school districts located in opposite ends of 
the state. One is a small urban district, while the other is a small rural district that is similar in 
size to many of the other rural districts within the state. Specifically, 6 schools (4 from the larger 
urban district, 2 from the smaller district) participated in the study. Collectively, the 
demographic characteristics of the teacher and student populations across the participating 
elementary schools are similar to the statewide elementary student and teacher population. 

Participants 

Participating classrooms (N = 39) enrolled 20 - 25 students, and all students were invited 
to participate in the project. Approximately 50% ( N = 486) of the students received parental 
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permission to participate in the study. As shown in Table 1, participants from the 20 classrooms 
randomly assigned to the intervention condition were slightly older and had a slightly higher 
percentage of students who were males, of minority racial status, received special education 
services, and had been retained. In addition to the student participants, 39 teachers (1 per 
classroom) also participated in the study. All of these teachers were Caucasian, and 79% were 
female. Most teachers reported significant classroom experience (M = 14.4 years, SD = 9). 

Intervention 

The SSIS-CIP (Elliott & Gresham, 2007) is a curriculum designed to teach students 10 
important social skills to support learning in the classroom. These skills are taught individually 
in units of three 20-25 minute lessons and include: listening to others, following directions, 
following classroom rules, ignoring peer distractions, asking for help, taking turns in 
conversations, cooperating with others, controlling anger during conflicts, acting responsibly, 
and showing kindness to others. Each lesson follows a 6-phase instructional model (coaching, 
modeling, role playing, practice/rehearsal, feedback from teachers, and generalization of skills 
through practice in other settings such as the lunch room or at home). 

Data Collection & Measures 

Child-level data were collected before and after SSIS-CIP implementation in the 
classrooms randomly assigned to the “treatment” condition. Specifically, two measures were 
used to assess key outcome variables related to students’ classroom behavior (Social Skills 
Improvement System Rating Scale, Cooperative Learning Observation Code for Kids). In 
addition, the Classroom Assessment Scoring System (CLASS, Pianta & Hamre, 2008) was used 
to assess the instructional environment in each participating classroom before and after treatment 
implementation. Finally, structured observations were used to assess fidelity of implementation 
of the SSIS-CIP curriculum in classrooms randomly assigned to the intervention condition. 

Social Skills Intervention System Rating Scales - Teacher Form (SSIS-RST; Gresham 
& Elliott, 2008). The SSIS-RST was completed by a child’s classroom teacher to assess 
participants’ social behavior in the classroom setting. The Social Skills scale of the SSIS-RST 
includes 46 items and yields seven subscales (Communication, Cooperation, Assertion, 
Responsibility, Empathy, Engagement, & Self-Control) in addition to the total composite. Each 
item is rated using a 4-point format ranging from Never to Almost Always. Psychometric 
evidence for scores from the SSIS-RST is strong and consistent with its intended purpose 
(Gresham & Elliott, 2008). Reliability estimates based on the current sample likewise is strong 
(Table 2). 

Academic Competence Evaluation Scales (ACES; DiPerna & Elliott, 2000). The ACES 
measures academic skills and academic enablers. The Motivation and Engagement subscales of 
the ACES were used for this study. The Motivation subscale includes 1 1 items that intend to 
measure a student’s approach, persistence, and level of interest regarding academic subjects. The 
Engagement subscale includes 8 items that reflect attention and active participation in classroom 
activities. Items are rated on a 5 -point format ranging from Never to Almost Always. Median 
internal consistency (.95) and test-retest stability (.83) are high for scores from the ACES. 

Cooperative Learning Observation Code for Kids (CLOCK; Volpe & DiPerna, 2010). 
The CLOCK is a structured observation format that was used to facilitate independent 
observations of student behavior in the classroom setting. The CLOCK features three categories 
of student behavior relevant to the primary research questions for this study: Positive Social 
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Behavior, Active Engaged Time (AET), and Passive Engaged Time (PET). Positive social 
encompasses any appropriate social behavior that is pennitted during the observation interval, 
and it was observed using a partial interval format with each interval lasting 15 seconds. AET 
and PET reflect student engagement in instruction, and momentary time sampling was used to 
record these behaviors. The CLOCK is based on a compilation of codes similar to other 
empirically supported classroom observation systems, such as the Behavioral Observation 
System of Young Students (BOYS; Volpe & Missal, 2007) and the Behavior Observation 
System for Students (BOSS; Shapiro, 1996). Each observation was 12 minutes, and one paired- 
observation (two raters) was completed per target student and data collection period. Agreement 
for these observations was high across all target behavior domains and paired observations 
(percentage agreement = .96 - .99, kappa = .91 - .99). 

Classroom Assessment Scoring System: Kindergarten - Third Grade (CLASS K-3; 
Pianta & Hamre, 2008). The CLASS K-3 is a structured observation system that was developed 
to assess the overall quality of the classroom instructional environment in the primary grades. 
Specifically, the CLASS K-3 yields scores in three domains: Emotional Support, Classroom 
Organization, and Instructional Support. These broad domains are further differentiated across 
10 dimensions (Positive Climate, Negative Climate, Teacher Sensitivity, Regard for Student 
Perspective, Behavior Management, Productivity, Instructional Learning Formats, Concept 
Development, Quality of Feedback, & Language Modeling). Each dimension is rated on a 7- 
point scale ranging from Low to High after an observer completes an observation “cycle” (20 
minutes of observation followed by 10 minutes of assigning ratings to dimensions/domains). 
Psychometric evidence for the CLASS is sound (Hamre & Pianta, 2008) and provides support 
for its intended purpose. In the current study, each classroom was observed once (2 cycles) 
during the first data collection window to detennine if there were significant differences in 
instructional environments across the participating classrooms. 

Research Design & Data Analysis 

This study used a Cluster Randomized Trial ( CRT) to test the efficacy of SSIS-CIP on 
each of the key outcome variables. Classrooms were randomly assigned to experimental 
conditions (SSIS-CIP and business-as-usual control) within six schools (see Figure 1). Multilevel 
modeling was used to evaluate the effects of SSIS-CIP to take into account students being nested 
within classes. We tested initially the degree to which the classes differ with respect to each of 
the outcomes of this investigation. These unconditional models yielded intraclass correlation 
(ICC) coefficients that detennined the degree to which the assumption of independence was 
violated due to the clustering of students in classes (Raudenbush, 1997). 

In evaluating effects of SSIS-CIP on each of the outcome measures, we included both 
student- and class-level predictors to adjust for their effects. Student-level predictors included 
pretest scores of the respective outcome measure (group-mean centered), students’ sex (l=male, 
0=female), race ethnicity (l=White, 0=other), and receipt of supplementary services (l=yes, 
0=no). The dummy variable predictors were grand-mean centered. Class-level predictors 
included grand-mean centered class average of pretest scores of the respective outcome measure. 
Treatment efficacy was tested using dummy codes for experimental conditions (1=SSIS-CIP, 
0=control). Moreover, interaction effects between treatment and pretest scores (both class- and 
student-levels) as well as student demographic variables (sex, race, and receipt of supplementary 
services) were tested by adding product terms between SSIS-CIP and each of the variables to the 
model. If the product terms were statistically significant at the .05 level, the pattern of interaction 
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was further examined by plotting the adjusted means. Otherwise, the non-significant product 
tenns were dropped from the final model for parsimony. We estimated multilevel models using 
the Mixed procedure of SAS (version 9.3) for teacher ratings of social skills. We used the 
Glimmix procedure for all classroom observation data. Because classroom observations 
consisted of frequency data that were highly skewed, we used Poisson distribution and log link 
for the Glimmix procedure. 

In addition, we estimated effect sizes of SSIS-CIP as compared to the control (business as 
usual) condition. Specifically, we computed the effect size as a standardized mean difference by 
dividing the adjusted (for pretest scores and other student- and class-level covariates) group 
mean difference by the unadjusted pooled within-group student-level standard deviation of the 
pretest outcome measure. This effect size computation (i.e., using student-level standard 
deviation to standardize the adjusted difference for Hedges’ g) followed the guidelines of What 
Works Clearinghouse for “ES computation based on results from HLM analyses in studies with 
cluster- level assignment” (WWC, nodate, p.45). Pooled within-group standard deviation of 
pretest scores was used because pretest scores were not affected by treatment. 

Findings / Results 

Student- and class-level means from the measures of social skills and academic 
engagement variables are reported in Tables 3 and 4. Parameter estimates for the final multilevel 
model for these variables are presented in Table 5. There was a statistically significant 
interaction between SSIS-CIP and class-level pretest on teacher ratings of all social skills and 
academic engagement/motivation measures except assertion and self-control. The adjusted 
differences between SSIS-CIP and control classrooms were larger for classes that had lower 
average pretest scores on these measures. For classes that had high average pretest scores, SSIS- 
CIP did not improve their average scores when holding other variables constant. Moreover, there 
was a statistically significant interaction between SSIS-CIP and student-level pretest on teacher 
ratings of academic motivation. 

Effect sizes were calculated for the social skills and academic motivation/engagement 
measures (Table 4) at the mean of their respective pretest scores and controlling students’ sex, 
race-ethnicity, special education, and supplementary service status. SSIS-CIP effect sizes on 
posttest teacher ratings ranged from .22 (self-control) to .38 (social engagement). The magnitude 
of these effect sizes might be considered small-medium according to Cohen’s (1988) criterion; 
however, as noted in the previous paragraph, these effect sizes should be interpreted carefully 
because the effectiveness of SSIS-CIP depended on class pretest of the respective outcome. 

Conclusions 

The purpose of this study was to evaluate the efficacy of the SSIS-CIP using a Multi-Site 
Cluster Randomized Trial (CRT). Based on the results of this study, the SSIS-CIP yields small- 
medium positive effects (increases) in prosocial behavior (overall, communication, cooperation, 
responsibility, empathy, social engagement, social skills intervention composite). In addition, 
SSIS-CIP positively impacted academic engagement and motivation, suggesting there may be 
academic benefits from its implementation. Across all of these variables, the effects of SSIS-CIP 
appear to be more specific to those children with more severe deficits in these areas prior to 
SSIS-CIP implementation. 
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Appendix B. Tables and Figures 


Table 1 

Student Demographic Characteristics by Condition 



SSIS-CIP 
A =262 

Control 
A =224 

Age (in years) 3 

7.43 (.40) 

7.30 (.37) 

Male 

45.21 

43.75 

White b 

66.67 

79.51 

Special education consideration 

5.63 

6.34 

Special education 

11.26 

6.34 

Supplementary services 

21.21 

26.83 

Retained in grade in prior year 

6.49 

3.90 

Promoted to next grade 

98.70 

100.0 


Note. Mean ( SD ) are reported for Age; % reported for all other variables. Race, special 
education, supplementary services, and retention/promotion were unavailable for approximately 
10% of cases. 

3 t-test result significant at .05 level; 
b chi-square result significant at .05 level. 
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Table 2 

Reliability and Intra-Class Correlation for Social Skills and Academic Engagement Measures 



Reliability Index 
Pretest Posttest 

Pretest 

ICC 

Posttest 

Social Skills 

Teacher Rating 3 





Social Skills Composite 

.98 

.98 

.28 

.26 

Communication 

.91 

.92 

.33 

.32 

Cooperation 

.92 

.93 

.13 

.16 

Assertion 

.85 

.87 

.37 

.31 

Responsibility 

.92 

.92 

.20 

.18 

Empathy 

.94 

.94 

.21 

.17 

Social Engagement 

.93 

.94 

.30 

.26 

Self-Control 

.93 

.95 

.22 

.22 

Classroom Observation b 





Positive Social 

.89 

— 

.34 

.34 

Academic Engagement 

Teacher Rating 3 





Motivation 

.98 

.98 

.13 

.11 

Academic Engagement 

.96 

.96 

.17 

.20 

Classroom Observation 11 





Active engaged time 

.93 

— 

.23 

.24 

Passive engaged time 

.90 

— 

.34 

.26 


Note. ICC= Intra-Class Correlation 
a Cronbach’s alpha 
b Kappa agreement index 
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Table 3 

Student-Level Means for Pretest and Posttest Social Skills and Academic Engagement by Treatment Condition 



Pretest 

Posttest 


SSIS 

Control 

SSIS 

Control 

Social Skills 

Teaching Ratings 

N=255 

N=224 

N=258 

N=221 

Social Skills Composite 

2.227 (.483) 

2.166 (.561) 

2.400 (.472) 

2.175 (.578) 

Communication 

2.354 (.529) 

2.295 (.614) 

2.522 (.489) 

2.305 (.613) 

Cooperation 

2.125 (.642) 

2.060 (.673) 

2.287 (.621) 

2.066 (.706) 

Assertion 

1.993 (.548) 

1.984 (.628) 

2.245 (.551) 

2.072 (.612) 

Responsibility 

2.322 (.572) 

2.255 (.644) 

2.457 (.550) 

2.233 (.642) 

Empathy 

2.220 (.559) 

2.243 (.655) 

2.382 (.552) 

2.195 (.674) 

Social Engagement 

2.329 (.557) 

2.204 (.618) 

2.510 (.517) 

2.202 (.652) 

Self-Control 

2.242 (.563) 

2.128 (.696) 

2.386 (.585) 

2.151 (.675) 

Social Skills Intervention Composite 

2.203 (.504) 

2.136 (.580) 

2.372 (.506) 

2.129 (.610) 

Classroom Observation 

N=1 19 

N=1 13 

N=120 

N=1 12 

Positive Social 

.496 (.636) 

.350 (.448) 

.370 (.573) 

.254 (.431) 

Academic Engagement/Motivation 

Teacher Ratings 

N=255 

N=224 

N=258 

N=221 

Motivation 

3.545 (1.038) 

3.381 (1.070) 

3.777 (.966) 

3.383 (1.125) 

Academic Engagement 

3.878 (.931) 

3.580 (.995) 

4.162 (.850) 

3.679(1.030) 

Classroom Observation 

N=1 19 

N=1 13 

N=120 

N=1 12 

Active Engaged Time 

.397 (.130) 

.368 (.139) 

.384 (.139) 

.364 (.149) 

Passive Engaged Time 

.340 (.123) 

.363 (.145) 

.355 (.141) 

.363 (.140) 
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Table 4 

Class-Level Means (SDs)for Pretest and Posttest Measures of Social Skills and Academic Engagement/ Motivation 
by Treatment Conditions 


Measures 

Pretest 

Posttest 

Adjusted 


SSIS (N=20) 

Control (N=19) 

SSIS (N=20) 

Control (N=19) 

Standardized 

differences 3 

Social Skills 

Teaching Ratings 

Social Skills Composite 

2.239 (.248) 

2.179 (.363) 

2.392 (.216) 

2.182 (.348) 

.36 

Communication 

2.380 (.279) 

2.309 (.428) 

2.520 (.246) 

2.314 (.413) 

.32 

Cooperation 

2.129 (.259) 

2.053 (.326) 

2.280 (.249) 

2.053 (.351) 

.29 

Assertion 

2.009 (.397) 

1.998 (.393) 

2.242 (.362) 

2.084 (.355) 

.26 

Responsibility 

2.330 (.296) 

2.268 (.341) 

2.450 (.239) 

2.244 (.336) 

.30 

Empathy 

2.229 (.232) 

2.267 (.391) 

2.369 (.205) 

2.208 (.367) 

.35 

Social Engagement 

2.351 (.291) 

2.214 (.407) 

2.502 (.249) 

2.206 (.373) 

.38 

Self-Control 

2.243 (.282) 

2.150 (.401) 

2.372 (.285) 

2.156(368) 

.22 

Social Skills Intervention Composite 

2.209 (.234) 

2.148 (.338) 

2.361 (.235) 

2.134 (.348) 

.36 

Classroom Observation 

Positive Social 

.492 (.451) 

.351 (.268) 

.370 (.405) 

.252 (.259) 

0 

Academic Engagement/Motivation 

Teacher Ratings 

Academic Motivation 

3.564 (.428) 

3.351 (.497) 

3.797 (.320) 

3.364 (.454) 

.34 

Academic Engagement 

3.901 (.444) 

3.589 (.481) 

4.148(350) 

3.674 (.529) 

.29 

Classroom Observation 

Active Engaged Time 

.396 (.072) 

.369 (.090) 

.384 (.080) 

.364 (.096) 

-.03 

Passive Engaged Time 

.340 (.075) 

.363 (.104) 

.355 (.085) 

.364 (.091) 

.10 


Note. — = not available. 

“Adjust for pretest scores and other student- and class-level covariates. 
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Table 5 


Mixed Model Estimates (Standard Errors) for Social Skills and Academic Engagement /Motivation Measures 


Intercept Pretest Pretest Condition Student Class Male White Special Supp. Intercept Residual 
(student) (class) pretest * pretest * Education Services variance variance 

Condition Condition 

Social Skills 


Teacher Ratings (N=434) 


Social Skills 

2 . 202 ** 

747** 

. 886 ** 

17 j** 

NA 

-.677** 

-0.055 

- 0.020 

-0.045 

-0.106** 

.026** 

.085** 

Composite 

(.043) 

(.036) 

(-127) 

(.061) 


(. 221 ) 

(.030) 

(.038) 

(.055) 

(.039) 

(.008) 

(.006) 

Communication 

2.333** 

.633** 

.880** 

.161* 

NA 

-.675** 

-.053 

-.027 

-.076 

- 126** 

.032** 

.113** 


(.048) 

(.038) 

(.118) 

(.068) 


(.214) 

(.034) 

(.043) 

(.064) 

(.045) 

(. 010 ) 

(.008) 

Cooperation 

2.093** 

.728** 

.995** 

.174* 

NA 

-.684* 

-0.061 

0.061 

-0.069 

-0.150** 

.032** 

.157** 


(.052) 

(.035) 

(-167) 

(.072) 


(.263) 

(.041) 

(.050) 

(.074) 

(.052) 

(. 012 ) 

(- 011 ) 

Assertion 

2.072** 

.648** 

.638** 

.153 

NA 

NA 

-0.119** 

-0.061 

-0.026 

-0.103* 

.049** 

.127** 


(.058) 

(.040) 

(.108) 

(.081) 



(.036) 

(.046) 

(.067) 

(.047) 

(.014) 

(.009) 

Responsibility 

2.271** 

.721** 

.915** 

.163* 

NA 

-.691** 

-0.062 

-0.041 

-0.116 

-0.127** 

.027** 

. 120 ** 


(.047) 

(.034) 

(-146) 

(.065) 


(.223) 

(.036) 

(.044) 

(.065) 

(.046) 

(.009) 

(.009) 

Empathy 

2.183** 

.650** 

.869** 

.192** 

NA 

-.767** 

- 0 . 111 ** 

-0.008 

0.045 

-0.097 

. 021 ** 

.162** 


(.045) 

(.040) 

(. 121 ) 

(.063) 


(.231) 

(.042) 

(.050) 

(.075) 

(.052) 

(.009) 

(. 012 ) 

Social 

2.255** 

.715** 

799 ** 

.206** 

NA 

-.482* 

-0.025 

-0.050 

-0.132 

-0.130** 

.030** 

.128** 

Engagement 

(.049) 

(.039) 

(-127) 

(.069) 


(.217) 

(.036) 

(.046) 

(.068) 

(.047) 

(. 010 ) 

(.009) 

Self-Control 

2.186** 

.731** 

. 744 ** 

.140 

NA 

NA 

-0.039 

-0.007 

-0.064 

-0.088 

.035** 

.136** 


(.052) 

(.034) 

(■ 111 ) 

(.073) 



(.037) 

(.047) 

(.069) 

(.048) 

(. 012 ) 

(. 010 ) 

Social Skills 

2.160** 

. 734 ** 

944 ** 

.180** 

NA 

-.637* 

-0.054 

-0.009 

- 0.022 

-0.123** 

.030** 

.105** 

Intervention 

(.047) 

(.036) 

(-147) 

(.066) 


(.255) 

(.033) 

(.042) 

(.061) 

(.043) 

(. 010 ) 

(.007) 

Composite 













Classroom Observations (N=202) 











Ln(Positive 

-1.378** 

.097 

1.226** 

.052 

NA 

NA 

-.371 

.061 

-.123 

-.039 

.070 

— 

social) 

(.215) 

(1.23) 

(.298) 

(-296) 



(.258) 

(.326) 

(.644) 

(.356) 

(.131) 
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Academic Engagement/Motivation 

Teacher Ratings (N=434) 


Academic 

Motivation 

3.443** 

(.065) 

..869** 

(.048) 

.810** 

(.124) 

.333** 

(.090) 

_ 179** 
(.060) 

-.577** 

(.196) 

0.045 

(.060) 

0.202** 

(.073) 

-0.047 

(■111) 

-0.182* 

(.080) 

.040** 

(.017) 

.341** 

(.024) 

Academic 

Engagement 

3.809** 

(.063) 

.698** 

(.036) 

.969** 

(.125) 

.267** 

(.087) 

NA 

-.490* 

(.184) 

-0.038 

(.057) 

0.005 

(.070) 

-0.003 

(.107) 

-0.188* 

(.075) 

.033* 

(.015) 

.317** 

(.023) 

Classroom Observations (N=202) 











Ln(Active 
engaged time) 

-1.006** 

(.169) 

-.002 

(1.122) 

1.927 

(1.461) 

-.012 

(.236) 

NA 

NA 

-.107 

(.242) 

-.031 

(.270) 

-.198 

(.527) 

-.024 

(.297) 

0 

— 

Ln(Passive 
engaged time) 

-1.041** 

(.172) 

-.003 

(1.250) 

1.707 

(1.429) 

.037 

(.241) 

NA 

NA 

-.051 

(.240) 

-.028 

(.277) 

-.006 

(.486) 

.047 

(.300) 

0 

- 


Note. NA=not included in the model because it was not a statistically significant predictor at the .05 level. * p < .05; ** p < .01. 
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Enrollment 


Figure 1. SSIS Classroom-Randomized Trial Student-Level Participant Flow Chart 
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